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ABSTRACT 

This paper makes recommendations for developing the next 
generation of state accountability systems for postsecondary vocational 
education (PVE). It focuses on the need to improve the core indicators for 
PVE; reduce the burden and improve the value of Carl D. Perkins Vocational and 
Applied Technology Education Amendments of 1998 (Perkins III) accountability 
requirements; and integrate future federal VE measures into a comprehensive 
institution-centered framework that integrates PVE and workforce development 
performance indicators. First, the paper summarizes major trends in the 
postsecondary accountability environment at national and state levels and 
presents how community and technical colleges are responding by creating 
performance indicator frameworks. Second, it reviews the major accomplishments 
and remaining problems and issues in the Perkins III effort and makes 
recommendations on how to build from these efforts. Third, the paper makes 
recommendations on developing and supporting core indicators for PVE and how 
they can be incorporated into broader institution-centered accountability 
systems for managing performance improvement. Finally, the paper makes 
recommendations for the federal role in accountability and how the Office of 
Vocational and Adult Education can play a catalyst role in improving state and 
institutional accountability systems for postsecondary education, including 
VE. (Contains 21 references.) (YLB) 
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Improving State Accountability Systems for 
Postsecondary Vocational Education 

Robert G. Sheets 



Introduction 

Since the 1980s, state community and technical college boards and agencies 
and institutions have experienced a major increase in federal and state accountability 
requirements. This increase has been accompanied by a shift in emphasis from access 
to performance. This new environment is prompting state boards and agencies and 
institutions to develop new performance indicator systems for reporting and managing 
continuous improvement for all major functions and programs, including vocational 
education. 

The Carl D. Perkins Vocational and Applied Technology Education Amendments 
of 1998 (Perkins III) have been a major driving force in creating this new accountability 
environment and in shaping state and institutional responses to it. The development of 
the Perkins III accountability system for postsecondary vocational education marked the 
first major federal-state effort to develop and fully implement a national framework for 
state accountability systems for vocational education in the United States. This effort 
broke new ground and improved the measurement and reporting of secondary and 
postsecondary student outcomes in vocational education. 

However, this effort also highlighted some major issues that should frame future 
discussions and efforts to improve postsecondary accountability systems. One major 
issue is whether the Perkins III core indicators are comprehensive and valid indicators 
for measuring the performance of postsecondary vocational education and whether 
states have the capacity to collect and report high quality data for these measures. A 
second issue is whether the Perkins III accountability requirements are creating 
unnecessary burden on states and institutions and how to improve their value and lower 
their costs. A third major issue is whether there should be more consistency between 
these measures and other federal, state, and institutional performance measures 
(OVAE, 2000b). Should the Perkins III core indicators be designed to fit within a larger 
postsecondary education framework that integrates vocational education measures with 
other postsecondary education and workforce development measures? 

The purpose of this paper is to make recommendations for developing the next 
generation of state accountability systems for postsecondary vocational education. This 
paper focuses on the need to: (1) make improvements in the core indicators for 
postsecondary vocational education, (2) reduce the burden and improve the value of 
Perkins accountability requirements, and (3) integrate future federal vocational 
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education measures into a comprehensive institution-centered framework that 
integrates postsecondary education and workforce development performance 
indicators. This paper also makes recommendations for the federal role in accountability 
and how the Office of Vocational and Adult Education (OVAE) can play a catalyst role in 
improving state and institutional accountability systems for postsecondary education, 
including vocational education. 

This paper first summarizes the major trends in the postsecondary accountability 
environment at the national and state levels and how community and technical colleges 
are responding by creating performance indicator frameworks. The paper then reviews 
the major accomplishments and remaining problems and issues in the Perkins III effort 
and makes recommendations on how to build from these efforts. Finally, the paper 
makes recommendations on developing and supporting core indicators for 
postsecondary vocational education and how they can be incorporated into broader 
institution-centered accountability systems for managing performance improvement. 

The Changing Postsecondary Accountability Environment for 
Community and Technical Colleges 

What are the major changes in the postsecondary accountability environment 
that are shaping the development and implementation of new accountability systems at 
the national, state, and institutional levels? How are state community and technical 
college systems and institutions responding? What are the implications for future efforts 
in developing federal accountability systems for postsecondary vocational education? 

Quality Assurance: From Self-Regulation to External Accountability 
Systems 

Throughout m ost of the 2 0 th century, federal a nd state g overnments I ooked to 
higher educational institutions to define their own mission and goals and to define the 
criteria for quality in higher education services (Bogue and Saunders, 1992). Higher 
educational institutions were also given the authority to regulate themselves by 
developing and managing their own quality assurance systems through internal 
management systems and participating in national and regional accreditation systems. 
Leading higher educational institutions and federal and state governments promoted 
and supported the establishment of national and regional accreditation organizations to 
assure the quality of postsecondary education. 

These self-regulation systems resulted in the development of internally focused 
quality assurance systems that were based on missions and goals defined by 
administrators and faculty members with only limited input from external stakeholders 
and customers. They also lead to the development of internally focused “quality 
indicators” that specified the organizational capabilities, conditions, practices, and 
resources necessary for p roviding h igh q uality p ostsecondary e ducational s ervices to 
students and fulfilling an institution’s mission and goals. These quality indicators 
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provided the basis for government-endorsed, peer-controlled accreditation systems that 
used institutional self-study and analysis followed by peer review and validation 
coordinated through national and regional accreditation organizations. 

Starting in the 1970s and 1980s, federal and state governments and other 
external stakeholders took a more active role in ensuring the quality of postsecondary 
education in the United States. The federal government added new reporting 
requirements and set performance requirements for student grant and loan programs. 
States expanded the roles of higher education boards and commissions in ensuring 
greater accountability. Both federal and state governments established new external 
accountability systems to ensure that higher education was addressing a broader set of 
public and consumer expectations for postsecondary education and were measuring 
and monitoring the fulfillment of these expectations. These new systems did not 
replace accreditation. Rather, they represented an additional layer of accountability. 

As noted by Levine (1992), this shift was prompted not by a well-documented 
“failure of performance” which was the driver in the K-12 accountability movement. 
Rather, it was a more generalized “crisis of confidence” and a growing recognition that 
federal and state governments were not taking an active role in defining the public 
interest in higher education and managing their sizeable investments. This shift was 
prompted by growing distrust of higher education and the perceived need to get a 
handle on the costs, productivity and effectiveness of higher education during a period 
of economic downturn and shrinking federal and state resources. There was a 
widespread and growing perception that traditional mechanisms for self-regulation and 
quality assurance were in disarray and could not be trusted. 

These federal and state initiatives to establish external accountability systems 
prompted a major shift in national and regional accreditation organizations to focus 
more attention on external stakeholder and customer requirements. They also 
prompted a shift from internal “closed system” to external “open-system” models for 
quality assurance within higher educational institutions, including community and 
technical colleges (Alfred, 1997). 

Performance Indicators: Expanding the Focus to Student Outcomes 

During the 1970s and 1980s, national and state accountability initiatives focused 
on developing comprehensive performance indicator systems to monitor the 
effectiveness of higher education (Borden and Bottrell, 1994). The early performance 
indicator systems focused on access, productivity, and quality with only limited attention 
to student outcomes. These initiatives focused on increasing access to higher 
education, especially minorities a nd underserved populations and in controlling costs 
and increasing productivity. Service output and cost measurement focused attention on 
how dollars were spent and justified the need for additional spending based on the 
quantity of services (e.g., units of instruction) provided and the additional costs of these 
services. During this same period, many states established institutional and program 
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effectiveness reviews that used “program quality” indicators along with market demand, 
student enrollment, and cost information to improve the quality and mix of 
postsecondary programs. These program quality indicators were similar to the quality 
criteria established by national and regional accreditation organizations. 

During the 1990s, national and state initiatives expanded the focus to measuring 
the performance of postsecondary educational institutions based in part on student 
outcome indicators. Higher educational institutions were increasingly held accountable 
not only for providing access to high quality programs, but a Iso for achieving results 
important to students and external stakeholders. Two major types of student outcomes 
were learning outcomes and economic outcomes. 

Learning Outcomes: Skill and Credential Attainment. Most early federal and 
state initiatives focused major attention on the success of students in attaining 
postsecondary degrees. Some state performance indicator systems also reported 
course completion rates with the assumption that degree or course completion was a 
valid and reliable indicator of success in achieving learning outcomes. Starting in the 
1980s, this assumption was called into question. The growing distrust of higher 
education was heightened by national studies and reports on lowered standards and 
grade inflation and the skill deficiencies of college graduates. Public and private leaders 
called for major reforms in postsecondary assessment and credentialing (National 
Governor’s Association, 2000). This resulted in m ajor n ational and state i nitiatives t o 
improve postsecondary assessment and credentialing systems so that they can provide 
valid and reliable documentation of student learning outcomes. This also has prompted 
national and regional accreditation organizations to focus on improving student 
assessment systems and promoting new “outcome-based accreditation” models for 
four-year and two-year higher educational institutions. This remains a major challenge 
for postsecondary education as exemplified by the recent National Report Card on 
Higher Education that gave all states a grade of “incomplete" on their performance in 
student learning (National Center for Public Policy and Higher Education, 2000). This is 
also exemplified by national initiatives by higher education and business organizations 
to improve the quality of credentials including postsecondary education degrees and 
certificates (National Alliance of Business, 2001). 

Economic Outcomes: Employment and Earnings. During the 1980s and 
1990s, the focus of higher education accountability efforts expanded beyond learning 
outcomes to include economic returns to students attaining postsecondary degrees and 
credentials. The beginnings of this shift can be traced to the early 1980s when 
postsecondary educational institutions justified the need for higher levels of federal and 
state investment because of their contributions to state and local economic 
development and the increased earnings of American workers (Ewell, 1994). This also 
can be traced to efforts by postsecondary education representatives to justify increased 
tuition costs by arguing that postsecondary education remained a “high-value” 
investment for students because of the increasing economic returns to postsecondary 
education in the form of higher wages and earnings. With increased federal and state 
funding and greater flexibility in increasing tuition costs, higher education promised to 
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promote state and local economic development and raise the earnings of college 
graduates. In the 1990s, public and private leaders, including governors and state 
legislators, began asking for evidence that postsecondary education was delivering on 
these promises. In response, state higher educational agencies and boards launched 
initiatives to address the economic returns to postsecondary education. These 
initiatives included the development of new performance indicators and systems to 
collect information on the employment and earnings of students receiving 
postsecondary degrees and credentials (Mundhenk, 2000; Seppanen, 1995). These 
efforts were focused on the professional degree programs of four-year colleges and the 
career and technical programs at two-year community and technical colleges. 

Increased Requirements for Consumer Information and Feedback 

During the 1980s and 1990s, most industries faced a consumer revolution. 
Consumers wanted more power in their purchasing decisions by having more 
information about products and services and the choices they were given in the 
marketplace. Consumer organizations and businesses responded by gathering and 
reporting comparative information on products and services and developed new 
materials and services to advise consumers. 

As the costs of higher education continued to rise throughout the 1980s and 
1990s and the value of higher education degrees increased in the labor market, this 
consumer revolution spilled over into the higher education marketplace. Parents, 
students, and public officials began demanding to know more about what they were 
getting for their money and they wanted the information to make comparisons between 
institutions and programs to improve their decisions. In response, federal and state 
governments focused on improving consumer information to help students chose 
colleges and universities with proven track records. 

Consumer Information Systems. Federal and state governments launched 
major consumer information initiatives to respond to this growing demand for consumer 
information in higher education. Federal higher education legislation established 
“student-right-to-know” requirements for higher educational institutions. Federal 
agencies and departments converted national reporting systems, such as IPEDS, into 
consumer information systems that provided comparative information on colleges and 
universities. In addition, the Workforce Investment Act (WIA) established strong 
consumer information requirements for training providers, which included many 
community and technical colleges. These government initiatives developed reporting 
and consumer information systems to provide accurate and reliable data in a wide 
variety of areas including program completion and employment and earnings. 

In addition, private organizations including leading magazines, such as U.S. 
News and World Report, began compiling and distributing comparative information on 
the “quality” and “value” of higher educational institutions and specific degree programs. 




5 7 



Improving State Accountability Systems for Postsecondary Vocational Education 



These private efforts have lead to the development of new measures for the quality and 
performance of higher education institutions. 

Customer Feedback: Measuring and Reporting Customer Satisfaction. This 
consumer revolution in conjunction with a related quality movement in higher education 
has lead to more emphasis on measuring and reporting customer satisfaction. The 
higher education community is debating how to measure and use student feedback 
(Kulik, 2001) and overall customer satisfaction in performance reporting and internal 
improvement efforts similar to leading public and private sector models such as Kaplan 
and Norton’s Balanced Scorecard and the Baldrige framework. This is also being 
debated in the implementation of WIA requirements for measuring customer 
satisfaction. 

From Performance Reporting to Performance Management: Financial 
Incentives and Continuous Improvement 

During the 1990s, federal and state agencies used their performance indicators 
systems to not only report performance d ata but to manage performance. This was 
done by providing financial incentives to promote improvements and developing 
performance management requirements that asked institutions to develop and manage 
improvement strategies and report progress on these efforts. 

Performance-Based Planning and Budgeting. Federal and state governments 
are undertaking major initiatives to use financial incentives and the budgeting process to 
drive performance improvement on a core set of performance measures. The 
performance incentive requirements in the WIA and Perkins III are only two examples of 
the use of performance incentives in federal legislation and federal agencies. In 
addition, states have been major players in using performance-based budgeting and 
financial incentives (Liner et al., 2001). In particular, states have taken major steps to 
use financial incentives and new approaches to planning and budgeting to promote 
performance improvement at the system and institutional levels in higher education 
including community and technical colleges (Ashworth, 1994). 

Performance Planning and Management. The Government Performance and 
Results Act (GPRA) required federal agencies to develop strategic plans that 
established performance goals and strategies and “results frameworks” to manage 
improvement of performance over multiple years. States also established similar 
strategic planning and performance management requirements for agencies. These 
new federal and state requirements have expanded the purpose of performance 
indicators systems from reporting to overall performance management. 
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The Response of Community and Technical College Systems: The 
Development of Performance Indicator Systems 

State higher education boards, including community and technical college 
boards, and colleges have responded to the new accountability environment by 
developing performance indicators systems. These indicator systems are designed to 
report performance to federal agencies and state legislatures and manage improvement 
efforts at the state and institutional levels. 

State higher education boards have developed comprehensive performance 
indicator systems for annual performance reports for state higher education systems. 
Some indicators apply to both four-year and two-year institutions (e.g., degree 
completion rates). Others are specifically designed to address the unique missions and 
roles of community and technical colleges. Community and technical college systems 
also have responded to the new accountability environment by developing 
comprehensive core indicator frameworks to measure and report “organizational 
effectiveness.” In 1994, the American Association of Community Colleges held a 
national roundtable to share information on leading practices in developing and using 
“core indicators of effectiveness.” This effort led to the development of national 
benchmarks for core indicators (Aired et al., 1999). 

The major performance areas and selected example measures from state and 
institutional core indicator frameworks are summarized in Figure 1 . These performance 
indicator systems reflect the multiple and distinct missions of community and technical 
colleges including their roles in: (1) academic preparation and transfer to four-year 
colleges; (2) the preparation for careers and employment; (3) adult, remedial and 
developmental education; (4) economic development and business and technology 
services; and (5) community development/service (Bailey and Averianova, 1998). 

These core indicator frameworks also reflect the highly diverse student 
populations served by community and technical colleges and the wide range of goals 
these students have in attending two-year colleges. Community and technical colleges 
have a unique role in higher education not only because of their different roles and 
functions but the students they serve. They are widely recognized as providing open 
access to a wide variety of non-traditional students who historically have not 
participated in postsecondary education. These students have a wide variety of goals 
that are different than traditional students. Many of these non-traditional students only 
want to gain a specific skill, not complete a program. Many of these students are 
already employed and are interested in changing jobs or career advancement. 
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Figure 1: Selected Indicators from State/Institutional Systems 
Student Access and Persistence 

• Number and percent of minorities and other targeted populations enrolling in 
postsecondary education 

• Student persistence (fall to fall enrollment) 

Skill and Credential Attainment 

• Skill attainment 

• Course completion 

• Degree/certificate attainment rate 

• Licensure/certification pass rates 

Student Transition 

• Placement rate in employment 

• Earnings at placement 

• Transition to four-year colleges 

• Retention at four-year colleges 

• Grade point averages/course grades of transitioned students 

Customer Satisfaction 

• Student goal attainment (self-reported) 

• Student satisfaction with services 

• Employer satisfaction with services 



Finally, these core indicator frameworks reflect the expanded definition of the 
customers and the growing recognition of employers and industry organizations as 
customers for economic and workforce development services. As community and 
technical colleges are expanding their role in economic and workforce development, 
they are increasingly addressing the needs- of employers and major industries in their 
service regions. 

Implications for Developing New Systems 

What are the implications of this changing postsecondary accountability 
environment and the responses of community and technical colleges for future efforts to 
develop federal performance accountability systems for postsecondary vocational 
education? The first implication is that future federal efforts should attempt to integrate 
multiple federal performance requirements a nd should be coordinated with state and 
local efforts to develop comprehensive frameworks for postsecondary institutions that 
balance the multiple missions and customer requirements of community and technical 
colleges. The second implication is that future efforts should attempt to improve the 
integration of federal and state performance measures with consumer information 
requirements. The third implication is that the focus should shift from federal reporting 
to performance management with an emphasis on using performance information and 
financial information to manage improvement and improve the return on public 
investment at the federal, state, and local levels. 
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Building from the Perkins III Core Indicator Framework: 

Problems and Issues 

The development of the Perkins III accountability system for postsecondary 
vocational education marked the first major federal-state effort to develop and fully 
implement a national framework for state accountability systems for vocational 
education in the United States. Perkins III made a major shift from Perkins II in the 
definition of the required indicators of performance moving from a gains to an 
attainment focus. OVAE worked with states for over a year to develop a core indicator 
framework that would provide the basis for addressing each of the required core 
indicators of performance while at the same time providing states with the flexibility to 
define their own measurement approaches and build from current state data collection 
and measurement systems. 

This section first describes the Perkins III core indicator framework and then 
addresses three major issues: 

• What should we be m easuring and how do we improve state and institutional 
capacity for collecting high quality data? 

• How do we reduce the burden and improve the value of the Perkins III 
accountability system for states and institutions? 

• How do we improve the consistency and integration of federal and state core 
performance indicators for postsecondary education and workforce 
development? 

Perkins III Core Indicator Framework 

Perkins III defines four major core indicators of performance, which provided the 
basis for developing the core indicator framework. 

(1) Student attainment of challenging State established academic and vocational 
and technical skill proficiencies. 

(2) Student attainment of a secondary school diploma or its recognized equivalent, a 
proficiency credential in conjunction with a secondary school diploma, or a 
postsecondary degree or credential. 

(3) Placement in, retention in, and completion of postsecondary education or 
advanced training, placement in military service, or placement or retention in 
employment. 

(4) Student participation in and completion of vocational and technical education 
programs that lead to non-traditional training and employment. 
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The Perkins III core indicator framework contains these core indicators for both 
secondary and postsecondary education. As shown in Figure 2, this framework 
established four core indicators and seven core subindicators for postsecondary 
vocational education. These subindicators are identified by a “P” next to each core 
subindicator. 

The core indicator framework defines the performance measures and 
measurement approaches for the fourteen subindicators. It also defines the quality 
criteria for these measurement approaches. As defined in the core indicator framework 
document (OVAE, 2000a), these are: 

• Performance Measures . The definition of the performance measures for each 
subindicator including the definition of the numerator and denominator of each 
performance measure. 

• Performance Measurement Approaches . The major state approaches for 
performance measurement for each performance measure. These approaches 
include assessment and data collection strategies. 

• Quality Criteria for Performance Measurement . The quality criteria for 
performance measurement to ensure sufficient rigor and comparability of state 
performance measurement and reporting. 



Figure 2: Perkins III Core Indicators and Subindicators 

Core Indicator 1 . Student Attainment 

1 SI : Secondary Academic Attainment 

1S2: Secondary Vocational and Technical Skill Attainment 

1 PI : Postsecondary Academic Attainment 

1 P2: Postsecondary Vocational and Technical Skill Attainment 

Core Indicator 2. Credential Attainment 

2S1 : Secondary Completion 

2S2: Proficiency Credential with Secondary Diploma 

2P1 : Postsecondary Degree or Credential 

Core Indicator 3. Placement and Retention* 

3S1 : Secondary Placement 
3P1 : Postsecondary Placement 
3P2: Postsecondary Retention 

Core Indicator 4. Participation in and Completion of Non-Traditional Programs 

4S1: Participation in Secondary Non-Traditional Programs 
4S2: Completion of Secondary Non-Traditional Programs 
4P1: Participation in Postsecondary Non-Traditional Programs 
4P2: Completion of Postsecondary Non-Traditional Programs 

* Placement and retention address employment, further education and/or military. 
Source: OVAE Core Indicator Framework (2000) 
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In the OVAE core indicator framework, states are held accountable for only those 
students with concentrated participation in vocational education — that is, those students 
who reach a state-defined threshold level of vocational education. Although states are 
required to report enrollment information for all vocational students, they are required to 
report performance information for only vocational concentrators (See Figure 3). Only 
two subindicators address a wider student population. These are the two subindicators 
for participation in non-traditional programs — 4S1 — Participation in Secondary Non- 
Traditional Programs and 4P1 — Participation in Postsecondary Non-Traditional 
Programs. These two subindicators address vocational participants — that is, students 
who enrolled in at least one vocational-technical education course. The placement and 
retention subindicators further restrict state accountability to only those vocational 
concentrators who successfully complete their, programs by receiving a credential. As 
shown in Figure 3, vocational completers are those vocational concentrators who have 
attained the academic and technical knowledge/skills/proficiencies in their 
programs/sequences of courses as evidenced by a diploma or degree or certificate. 

In using the core indicator framework, states have considerable latitude in 
defining their measures and measurement approaches. They also have considerable 
flexibility in how they define vocational concentrators and completers for reporting their 
results. 



Figure 3: Student Population Definitions 
Threshold Level of Vocational Education 

A threshold level of vocational education is defined as a program/sequence of courses or instructional 
units that provides an individual with the academic and technical knowledge/skills/proficiencies to prepare 
the individual for employment and/or further/advanced education (Section 3 (29) Definitions). 

Vocational Participant 

Student who enrolled in at least one vocational-technical education course. 

Vocational Concentrator 

Student who enrolled in a threshold level of vocational education. 

Vocational Completer 

Student who attained the academic and technical knowledge/skills/proficiencies within a 
program/sequence of courses or instructional units that provides an individual with the academic and 
technical knowledge/skills/proficiencies to prepare the individual for employment and/or further/advanced 
education. 

Source: OVAE Core Indicator Framework (2000) 
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Core Postsecondary Indicators: What Should We Be Measuring and 
How Can We Improve Data Quality? 

Although the development of the Perkins III core indicator framework was 
successful in supporting state efforts to report performance on all core indicators, this 
effort also raised some major problems and Issues for postsecondary accountability 
systems. This framework attempted to develop parallel secondary and postsecondary 
subindicators emphasizing consistency of measurement for schools and colleges. As 
found in OVAE’s state pilot project (OVAE, 2000), this effort resulted in major 
challenges for states in applying some measures to postsecondary institutions such as 
academic attainment. As seen in the discussion of current trends in states in 
developing “core indicators of effectiveness," this also has resulted in some major gaps 
in measuring critical outcomes addressed in state and institutional systems, such as 
earnings of completers and student satisfaction and accomplishment of student goals. 

The question of what we should be measuring can best be addressed by 
reviewing the development of each Perkins III core indicator and comparing the Perkins 
III subindicators to the state and institutional “core indicators of effectiveness” 
summarized in Figure 1 . 

Core Indicator 1: Student Attainment. The effort to create parallel measures 
for secondary and postsecondary education on academic attainment poses some major 
problems and issues. The core indicator framework was designed to build on and 
reinforce state academic standards and assessment systems designed for all 
secondary students including vocational education students. This was done to reinforce 
the commitment that vocational students should be expected to meet the same state 
academic requirements as all students and perform at or above the level of all students 
on state academic assessments. This effort was very timely and successful in improving 
secondary accountability but presents problems at the postsecondary level. One 
reason is that there is no equivalent for state academic standards at the postsecondary 
level and not all programs define separate academic requirements. Another reason is 
that students have varying needs and interests and may not take courses or programs 
with well-define academic requirements other than basic skill requirements. 

Another p roblem i s the I ack of strong m easurement approaches for vocational 
and technical skill attainment. Most states utilized existing institutional grading and 
credentialing systems to measure skill attainment with the argument that they are valid 
and reliable indicators of skill attainment. Many states used the same measurement 
approach — program/degree completion — for both skill attainment and Core Indicator 
2 — Credential Attainment— with the argument that these two indicators were measuring 
the same student outcome (OVAE, 2000b). These arguments are reasonable but may 
not stand the test of time in light of the growing distrust of postsecondary assessment 
and credentialing practices and growing calls for reform in assessment and 
credentialing at all levels. Also, most students do not enroll in degree and certificate 
programs but take courses to upgrade skills and transfer to other institutions. The 
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separate skill attainment measure has the potential to capture outcomes for these 
students. 

Recommendations . Future efforts should reconsider the use of a separate 
academic attainment measure. These efforts should focus on improving the 
measurement of vocational and technical skill attainment by improving the specification 
of learning outcomes and the assessment and credendialing of these skills by 
community and technical colleges. 

Core Indicator 2: Credential Attainment. This is probably the most widely 
accepted and least controversial core indicator for postsecondary education. This is 
because it reinforces and supports the widespread collection of this information to meet 
a variety of federal and state accountability requirements. However, state measurement 
approaches raise some major problems and issues. 

The first issue is what degree and certificate programs should be included. As 
reported in the OVAE state pilot project (OVAE, 2000b), some states include all AA/AS 
degree programs and all certificate programs; Others only include AS degree 
programs. This restricted interpretation fails to account for the fastest growing 
credentialing market for two-year colleges — short-term certificates. 

The second issue is what students should be included. Some states include all 
students that completed a threshold level of courses within a program and/or registered 
the intent to complete the program. Other states use approaches from other federal 
reporting requirements that only include a small subset of students who were first-time, 
full-time students. This substantially reduces the number of students reported and fails 
to capture the growing non-traditional student population at two-year colleges — adults 
who attend part-time and work on their programs in irregular patterns. As 
recommended by Aired et al. (1999), the measurement of credential attainment should 
represent a broader student population and then be reported by different student 
populations as necessary. 

Recommendations . Future efforts should focus on gaining broad state consensus 
on including the full range of degree and certificate programs and including all students 
who reach a threshold level regardless of whether they are first-time or returning 
students or whether they are full-time or part-time. As discussed above, future efforts 
also should focus on improving the specification and assessment of learning outcomes 
that are the basis for awarding degrees and certificates. 

Core Indicator 3: Placement and Retention. The Perkins III framework 

restricts the focus of placement and retention to measuring only the transition of 
students to further education, employment, and/or the military. The only exception is 
employment, which includes a measure of employment retention. The subindicators do 
not address whether students are successful at four-year institutions. And, they did not 
address the quality of employment as measured by wages and earnings. 
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As discussed earlier, many state and institutional core indicator systems not only 
address transition to four-year colleges but also measure the success of students after 
transition. They use a variety of measures including persistence, grade point averages 
compared to now transfer students, and the percent of credits successfully transferred. 
Many states also address the quality of employment including training-related 
employment a nd earnings of graduates. T he m easurement of employment retention 
and earnings is a critical linkage to workforce development including connections to the 
WIA performance measures. 

Another major issue is the different capacities of states to actually collect valid 
and reliable data on student transitions. Some states have developed the capacity to 
measure student transitions from community and technical colleges to four-year 
colleges and employment within and outside the state. However, these state capabilities 
vary tremendously by state. The result is wide variations in state performance due 
mainly to wide variations in data quality, not program quality. States need further 
support at the federal level to improve these systems (National Postsecondary 
Education Cooperative, 2001 ). 

Recommendations . Future efforts should expand the measurement of transition 
to include student retention and success at four-year colleges. Future efforts also 
should develop consistent measures with workforce development to address 
employment retention and earnings. Finally, future efforts should focus on improving 
state student transition data systems to improve the data quality for measuring transition 
to further education and employment. 

Core Indicator 4: Non-Traditional Participation and Completion. This core 
indicator is the only indicator focused on specific programs and a specific student 
population in vocational education — non-traditional students. This population is also a 
special population with states required to report performance on this population for all 
other core indicators. 

This core indicator raises some major questions and issues. The first issue is the 
wide variation in how states define non-traditional programs using their own systems for 
defining “programs.” (OVAE, 2000b) The second issue is whether postsecondary 
education should focus on participation in one course for 4P1 or focus more on actual 
enrollment in a non-traditional program, which requires more concentration, usually 9-12 
credit hours. 

However, the biggest question is the need for this core indicator and whether it 
should play such a large role in the performance accountability system. Very few critics 
would argue that the Perkins III system should not focus on improving the access of 
males and females to non-traditional programs that result in improved success, in the 
labor market. The question is: what is the overall outcome we want and what should we 
measure? Should the focus be on one strategy to improve equity in opportunity and 
earnings or should the focus be on measuring whether males and females are getting 
similar results? If so, then states and institutions should be held accountable for equity 
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in results and should use non-traditional careers as one strategy in closing the 
employment and earning gaps of males and females. 

Recommendations . Future efforts should focus on reporting the performance of 
demographic groups and special populations on getting equitable results in skill 
attainment, credential attainment, and further education and employment transitions. 
One major special population should be students (males and females) enrolling in non- 
traditional programs. Future efforts should then review the need to retain this core 
indicator as a separate stand-alone indicator for postsecondary education. 

Additional Core Indicators: Student Goal Attainment and Customer 
Satisfaction. In comparing the Perkins III core indicator framework with state and 
institutional efforts to develop their own core indicator frameworks, there is one major 
difference — the measurement of individual student goal attainment and student and 
employer satisfaction. Given the diverse needs and requirements of students attending 
community and technical colleges and the growth of enrollment in short-term, 
individualized education and training, this requires further exploration. In addition, the 
new emphasis on customer satisfaction in higher education and government also 
suggests the need for exploring the use of customer satisfaction to measure both 
student goal attainment and overall satisfaction with services. 

Recommendations . Future efforts also should explore the use of customer 
satisfaction indicators to address the attainment of student goals and the satisfaction of 
students and employers with postsecondary vocational education services. 

How Do We Reduce Burden and Improve Value? 

The second question is how to reduce the burden and improve the value of the 
Perkins III accountability system for states and institutions? 

As described earlier, state community and technical college boards and colleges 
themselves are facing a new accountability environment at the national, state and local 
levels. Performance reporting is now considered a normal cost of doing business. The 
question of burden is not one of simply being required to be accountable for results and 
be expected to report high quality performance data to external stakeholders and 
funders. The question of burden is raised in the current environment only when 
reporting requirements from different funders and stakeholders: 

(1 ) Are not valid indicators and steer states and institutions away from performance 

measures that matter, 

(2) Require too much additional requirements for reporting detailed performance 

information by multiple student populations, 
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(3) Are inconsistent with the reporting requirements from other funders and 
stakeholders requiring multiple data collection and reporting systems, 

(4) Do not provide sufficient funding or support to justify and offset the additional 
costs of doing business, and 

(5) Do not add value in improving the effectiveness of the organization in meeting 
stakeholder and customer requirements. 

Perkins III Core Indicators: Valid Indicators for Community and Technical 
Colleges? The question is not whether there are too many or too few Perkins 
measures. The question is whether they are the right measures and address the most 
critical goals of community and technical colleges in vocational education. Do the 
required measures provide a complete (scope) and accurate picture of the performance 
of colleges in vocational education? Are they the measures that really matter? 

As discussed earlier, most of the Perkins III core indicators match core indicators 
that states and institutions currently define and measure in their own performance 
indicator systems because they are the same indicators identified by institutions 
themselves and requested by other federal and state funders and stakeholders. As a 
result, they do not create unnecessary burden on states and institutions. However, as 
recommended above, other Perkins III measures should be reconsidered and, in some 
cases, eliminated. This will not necessarily reduce the number of core indicators 
because, as recommended above, future efforts should also consider additional 
measures including transition success, earnings, and student goal attainment and 
customer satisfaction. 

Additional Reporting Requirements: Need for Demographic Group and 
Special Populations Performance Data. Perkins III places requirements on states to 
report performance by demographic group and special population. As discussed 
earlier, many federal and state accountability systems require similar reporting and 
there is major interest in postsecondary education on equity in results such as the 
participation and success of minorities in postsecondary education. Future efforts 
should focus on developing standard definitions and measures of student 
characteristics to reduce burden. 

Consistency with Other Federal and State Reporting Requirements: Need 
for Aligned Postsecondary Education and Workforce Development Measures. As 

stated above, the real burden in the new accountability environment is not simply in the 
reporting of outcomes, but in the need to establish separate reporting systems for 
multiple federal agencies and state boards. As discussed above, one major focus in 
future efforts should be to increase the consistency of performance indicators and data 
collection requirements across federal agencies and states. 
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Reporting Requirements Backed by Funding/Technical Support: The Need 
for Building Capacity in Assessment and Student Transition Data Systems. 

Another major focus in future efforts should be providing states and institutions with the 
resources and technical support to gather and report high-quality data on the most 
critical core indicators. As discussed earlier, the most critical data quality problems are 
in skill attainment and transition measurement. Future efforts should focus on 
improving assessment and credentialing systems at postsecondary education 
institutions and improving state student transition data systems. 

Adding Value: Moving from Performance Reporting to Performance 

Management for Continuous Improvement. Future efforts should not only reduce 
costs but also improve value by improving state and institutional capacity to use data to 
improve performance. These efforts should focus on promoting best practices and 
models for managing improvement. They also should establish a national benchmarking 
system that provides states and institutions with comparative performance information. 

How Do We Improve the Consistency and Integration of Performance 
Indicators? 

The third question is how to improve the consistency and integration of 
performance indicators? 

In the new accountability environment, states and institutions are facing 
performance requirements from all sides. Performance measurement and reporting is 
now considered a normal cost of doing business in the education arena. As argued 
above, the real burden is having to address inconsistent or contradictory performance 
goals and having to develop and maintain multiple reporting systems for multiple 
federal, state, and local stakeholders. Although this problem is a general problem faced 
by all postsecondary education, it has the most severe impacts on community and 
technical colleges. This is because of their multiple missions and diverse customer 
goals and the difficult challenges of integrating and managing their multiple functions 
with different external stakeholders and performance requirements. This also is 
because community and technical colleges span two worlds — postsecondary education 
and workforce development. They are the major providers of education and training 
services for state and local workforce development systems funded by the Workforce 
Investment Act (WIA) and must balance and integrate both postsecondary and 
workforce development performance measurement and reporting requirements. 

Problems and Issues in Previous National Efforts. The call for consistent 
performance measures is nothing new. The need for common or consistent 
performance measures in postsecondary education and workforce development has 
been widely recognized for almost twenty years. Federal agencies and states have 
worked together over many years to align federal performance measures and reporting 
requirements (Baj, Sheets, and T rott, 1 994). 
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The first problem faced by previous efforts was the lack of educational 
performance measures. These efforts were hampered by the lack of a well-defined set 
of performance measures for postsecondary education, including vocational education, 
which could be integrated with workforce development measures. This is no longer the 
case. The problem now is too many different sets of postsecondary educational 
measures coming from many different units and programs in the U.S. Department of 
Education and other federal agencies. 

A second problem with previous efforts has been their federal-centered, systems- 
building approach. Previous efforts have focused on aligning existing federal measures 
with only limited attention to state and local performance requirements. This is because 
previous efforts were driven by the workforce development side. They were focused on 
developing comprehensive workforce development systems. Community and technical 
colleges are funded primarily at the state and local levels and are increasingly driven by 
state and I ocal performance requirements that address a II of their m issions i ncluding 
academic transfer, adult education, and economic development. They also are 
expected to address the needs and requirements of their regional economies and 
communities. Future efforts to integrate measures should balance federal, state, and 
local performance requirements. 

A third problem with previous efforts has been the lack of attention to how 
measures must be designed to promote improved performance across functional areas 
at the service provider level — in this case, at the institutional level. The assumption has 
been that if we develop consistent measures across funding streams for each separate 
measurement area— placement, skill attainment, program completion-then we have 
solved the problem. However, leading efforts in the private sector go a step farther 
(e.g., Kaplan and Norton, 1996, Eckes, 2001). They attempt to develop performance 
measurement systems that promote the integration and synergy of different functional 
areas and units by looking at the interrelationship between measures and whether they 
are having unintended negative consequences for the entire organization. They also 
develop strategic or balanced scorecard systems that can help the entire organization 
manage improvement strategies and manage whole end-to-end processes that provide 
value to customers. These systems help manage the performance of the entire 
enterprise from the perspective of the customer at the point of service delivery. 

Moving from Federal-centered to Institution-centered Frameworks. Future 
efforts should build from previous federal-centered efforts, but they should create more 
balance between federal, state and local performance requirements. They also should 
develop new performance indicator frameworks from multiple perspectives, not simply 
the perspective of federal agencies managing their own performance accountability and 
reporting systems. They should balance this federal perspective with state and local 
perspectives. However, the major focus should be the perspective of service providers. 
Future efforts should develop common or consistent measures, in part, from a 
comprehensive postsecondary education framework based on the perspective of 
postsecondary educational institutions and how they are expected to use these 
measures to manage performance improvement. This framework should then be used 
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by federal and state governments to develop their performance measurement 
requirements. This will result in an institution-centered framework that will create more 
value and add less burden for community and technical colleges. It will also have the 
best chance of being fully implemented and achieving returns on federal and state 
investment. 

Conclusions and Recommendations 

As discussed in this paper, there are growing national and state demands for 
accountability in postsecondary education. This new environment is prompting federal 
agencies, state boards and agencies, and institutions to develop new performance 
indicator systems for reporting and managing continuous improvement for all major 
functions and programs including vocational education. What role should the U.S. 
Department of Education play in the future and how should the department build from 
the accomplishments of the Perkins III effort? 

Federal Role in Postsecondary Accountability 

The federal government is a major funder and stakeholder in postsecondary 
vocational education through Perkins funding and other federal funding sources (e.g., 
student grants and loans) that are increasingly used by students attending community 
and technical colleges. The federal government has a legitimate role to play in ensuring 
that states and institutions are meeting the needs of targeted customers and providing a 
return on federal investment. 

However, the federal government has unnecessarily increased state and local 
accountability burdens by not developing consistent performance indicators across 
multiple federal programs and working with states and institutions to develop a common 
performance indicator framework that can be used to promote improvement in 
postsecondary education including vocational education. There is a need to integrate 
the Perkins III secondary core indicator framework within state secondary educational 
accountability systems. And, there is also a need to do the same thing at the 
postsecondary level by integrating future efforts into larger efforts to develop a core 
indicator framework for postsecondary education. This effort should place a strong 
emphasis on integrating postsecondary education and workforce development. 

To address this problem, the federal government should play a catalyst role in 
working with states and institutions to develop a national (not federal) consensus on an 
institution-centered core indicator framework for two-year colleges that address the full 
range of goals and functions of these unique institutions and address the needs of all 
funders and stakeholder, not just the federal government. This framework should 
address both performance indicators and student populations. The federal government 
should then use this framework to develop federal performance indicators and 
measures and data reporting requirements consistent with this framework. 
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The federal government also should play a major role in expanding the capacity 
of states and educational institutions to collect and use data to report performance and 
manage continuous improvement. The federal government should focus on improving 
assessment and credentialing practices and improving the capacity of states and 
institutions to measure student transitions. Also, the federal government should work 
with states and institutions to improve data management and analysis capacity at the 
state and local levels. 

The federal government also should play a catalyst role in shifting the focus from 
reporting performance to using performance information to manage continuous 
improvement. This should be done through two strategies. First, the federal 
government should work with states and institutions to develop data and technical 
standards for information systems and promote best practices in state and local 
information systems. This should provide the basis for improved federal reporting. 
Second, the federal government should promote the use of data for performance 
improvement by promoting best practices in performance management and continuous 
improvement and encouraging the development of a national benchmarking data 
system for states and institutions to use in obtaining comparative performance 
information. 

Building from the Perkins III Effort 

As discussed earlier, the development of the Perkins III accountability system for 
postsecondary vocational education broke new ground and improved the measurement 
and reporting of core secondary and postsecondary student outcomes in vocational 
education programs. It also used core indicators that are widely used by states and 
institutions in developing core indicators of effectiveness. 

Future efforts should build on the Perkins III accomplishments by integrating 
major elements of this framework into a larger institution-centered postsecondary 
framework. This effort should address some of the major problems including: (1) data 
quality problems in skill attainment measurement, (2) including all students and 
programs in credential attainment measurement, (3) expanding transition measurement 
to include success in four-year educational programs and earnings and improving state 
student transition data systems, and (4) recasting the non-traditional performance 
measure. These efforts also should explore adding additional measures to capture the 
achievement of student goals and customer satisfaction. 
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Major Recommendations 

1. OVAE and states should establish a national taskforce to develop a 
comprehensive institution-centered core indicator framework that two-year 
community and technical colleges can use to manage continuous improvement, 
provide consumer information, and report progress to external stakeholders 
through federal and state performance accountability systems. This framework 
minimally should establish a core set of performance indicators and student 
populations. 

2. This national taskforce should focus attention on the development of core 
indicators in two major areas: (1) skill and credential attainment and (2) transition 
to further education and employment. It should explore the development of new 
transition measures that address the success and retention of students 
transferring to four-year colleges and universities. It a Iso should explore new 
transition measures that address employment retention and earnings consistent 
with related measures in workforce development. 

3. This national taskforce should sponsor a national roundtable discussion on best 
practices in the measurement and use of customer satisfaction data in the public 
and private sectors with a special focus on how corporations use customer 
satisfaction data in their performance management systems. The national 
taskforce should use this forum to consider additional core indicators for 
customer satisfaction that can address student goal attainment and overall 
student and employer satisfaction with services. 

4. This taskforce should also focus on developing common definitions for all major 
student populations for reporting performance on the core performance 
indicators. The taskforce should focus on defining the major demographic 
groups and special populations. 

5. OVAE and its federal agency partners and states and national associations 
should use this core indicator framework to promote consistency in performance 
measurement requirements in federal and state legislation and consistency 
between the performance i ndicators d eveloped a nd u sed by federal a nd state 
agencies and programs. 

6. OVAE and states should focus future data quality efforts on two major areas: (1 ) 
assessment and credentialing systems used by postsecondary institutions to 
award degrees and certificates and (2) transition measurement especially the 
capacity of states to use administrative record exchange systems to measure 
student transitions to four-year colleges and universities and Ul-covered 
employment both within and between states. 
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7. OVAE and states should identify, develop, and promote models for institution- 
based performance management systems for managing continuous 
improvement. 

8. OVAE and states along with representatives from community technical colleges 
should launch a national project to use this institution-based performance 
indicator framework and leading performance management models as the 
foundation for establishing a comprehensive set of national (not federal) 
voluntary data content and technical standards for state and institutional 
information systems for the collection and distribution of information. 

9. OVAE and states should use these voluntary national data content and technical 
standards to develop federal and state reporting requirements and to develop a 
national benchmarking data system that states and institutions can use to 
benchmark their performance and manage their improvement efforts. 




22 



References 



Aired, Richard (1997), “From Closed to Open Systems: New Designs for Effectiveness 
in Community Colleges," Journal of Applied Research in the Community Colleges . 5(1): 
9-15. 



Aired, Richard, Peter Ewell, James Hudgins (1994), Core Indicators of Effectiveness for 
Community Colleges . Washington, D.C.: American Association of Community Colleges, 
Community College Press. 

Ashworth, Kenneth H. (1994), “Performance-Based Funding in Higher Education: The 
Texas Case Study,” Change . November/December 1994, pp. 9-15. 

Baj, John, Robert G. Sheets, and Charles E. Trott (1994), Building State Workforce 
Development Systems Based on Policy Coordination and Quality Asssurance . 
Washington, D.C.: National Governors’ Association. 

Bogue, E. Grady and Robert L. Saunders (1992), The Evidence for Quality: 
Strengthening the Tests of Academic and Administrative Effectiveness . San Francisco, 
CA: Jossey-Bass Publishers. 

Borden, Victor M.H. and Karen V. Bottrill (1994), “Performance Indicators: History, 
Definitions, and Methods,” in Victor M.H. Borden and Trudy W. Banta (eds.), Using 
Performance Indicators to Guide Strategic Decision Making . New Directions for 
Institutional Research, San Francisco, CA: Jossey-Bass Publishers, pp. 5-22. 

Eckes, George (2001), The Six Sigma Revolution, New York: John Wiley and Sons. 

Erwin, T. Dary (1999), “The United States Perspective on and Experience of 
Performance Indicators and Threshold Standards — How Is Quality Determined?” 

In Helen Smith, Michael Armstrong, and Sally Brown (eds.), Benchmarking and 
Threshold Standards in Higher Education . London: Kogan Page. 

Ewell, Peter (1994), “A Matter of Integrity: Accountability and the Future of Self- 
Regulation,” Change, November/December 1994, pp. 25-29. 

Kaplan, Robert S. and David P. Norton (1996), The Balanced Scorecard: Translating 
Strategy into Action . Boston, MA: Harvard Business School Press. 

Kulik, James A. (2001 ), “Student Ratings: Validity, Utility, and Controversy," in Michael 
Theall, Philip C. Abrami, Lisa A. Mets (eds.), The Student Ratings Debate: Are They 
Valid? How Can We Best Use Them?, New Directions for Institutional Research, San 
Francisco, CA: Jossey-Bass Publishers, pp. 27-44. 

Levine, Arthur (1992), “Why Colleges Are Continuing to Lose the Public Trust,” 

Change . July/August, 1992, p. 4. 




23 2 5 



Liner, Blaine, Harry P. Hatry, Elisa Vinson, Ryan Allen, Pat Dusenbury, Scott Bryant, 
Ron Snell (2001), Making Results-Based State Government Work . Washington, D.C.: 
The Urban Institute. 

Mundhenk, Robert T. (2000), “Institutional Effectiveness and Unemployment Insurance 
Data,” White Paper of the American Association of Community Colleges. 

National Alliance of Business (2001 ), Six Principles of Certificate Quality: The Employer 
Perspectives . Washington, D.C.: National Alliance of Business. 

o 

National Center for Public Policy and Higher Education (2000), Measuring Up 2000: 

The State-Bv-State Report Card for Higher Education . Washington: D.C.: National 
Center for Public Policy and Higher Education. 

National Governors’ Association (2000), Higher Expectations: Essays on the Future of 
Postsecondarv Education . Washington, D.C.: National Governors’ Association. 

National Postsecondary Education Cooperative (2002), Student T ransition Data 
Systems . Report of the Working Committee. 

Office of Vocational and Adult Education, U.S. Department of Education (2000a), 
Perkins III Core Indicator Framework . 

Office of Vocational and Adult Education, U.S. Department of Education (2000b), 
Postsecondarv Pilot Project Report . 

Seppanen, Loretta J. (1995), “Linkages to the World of Employment,” in Peter T. Ewell 
(ed), Student Tracking: New Techniques. New Demands. New Directions for 
Institutional Research. San Francisco, CA: Jossey-Bass Publishers, pp. 77-91). 




24 



26 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 



This document is covered by a signed "Reproduction Release (Blanket)" 
form (on file within the ERIC system), encompassing all or classes of 
documents from its source organization and, therefore, does not require a 
"Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may be 
reproduced by ERIC without a signed Reproduction Release form (either 
"Specific Document" or "Blanket"). 



ERIC 



EFF-089 (1/2003) 



k 



